Overview

Analyze eye glance behavior from a simulator study, where participants conduct ‘drives’ across grade crossings.
The crossings can have one of three types of errors:
- Gate, No Train
- Train, No Gate
- Stuck
In addition, the Emergency Notification Sign (ENS) can be placed parallel to the train tracks or perpendicular to the train tracks.
- Gate, No Train: Parallel or perpendicular.
- Train, No Gate: Parallel or perpendicular
- Stuck: Parallel or perpendicular
The combination of Error Type and Sign Position generates 5 possible conditions. Each participant experiences all 5 conditions, with the final condition always being ‘Stuck’ (parallel or perpendicular.)
- Each participant experiences all four conditions of Gate, No Train (parallel and perpendicular) and Train, No Gate (parallel and perpendicular), as well as one Stuck condition (parallel or perpendicular).
- The order of the first four conditions is randomized, while the last is always Stuck.

There are two possible outcomes:

Glanced: Participant correctly glanced at the sign
Called: Participant stated that they would make a call to the appropriate emergency number on the sign.

These will be analyzed as separate response variables, since it is possible to have one and not the other. Thus, we’ll have two models to report results on in the end:

\[ Glanced = \beta_1*Error Type + \beta_2*Sign Position + \beta_3*Age + \beta_4*Gender + \epsilon \]

And

\[ Called = \beta_1*Error Type + \beta_2*Sign Position + \beta_3*Age + \beta_4*Gender + \epsilon \] These will be logistic models, since the outcome variable in each case is binary. We’ll use model selection to decide if the mediating variables of age and gender are useful. In addition, we may consider the order that the scenarios were run as well, namely ‘first parallel’ and ‘first perpendicular’.

Variables

From study metadata:

Variable	Explanation
Participant ID	1 thru 48
Age	Participants Age
Gender	Participants Gender (M, F, TGNC)
Crossing ID	Run 1 = 1 (parallel), Run 2 = 2 (parallel), Run 3 = 3 (Perpendicular), Run 4 = 4 (Perpendicular), Run 5 = 5(Stuck on Track)
Error Type	Type of Malfunction (Gate, No Train or Train, No Gate)
Sign Position	Position if ENS (Parallel or Perpendicular)
Glanced	0 = no, 1 = Yes
Called	0 = no, 1 = Yes

EDA

Quick exploratory data analysis to check for data completeness and understand distributions.

library(readxl)
library(tidyverse)
library(knitr)
library(kableExtra)
library(ggplot2)
library(plotly)
library(sjPlot)

d <- read_xlsx(file.path('Data', 'Analysis and Glance Score 3 17 20.xlsx'), sheet = 'Data')

# Filter out rows where glanced or called is NA
d <- d %>% 
  filter(!(is.na(Glanced) | is.na(Called)))

summ_tab <- d %>%
  group_by(`Error Type`, `Sign Position`) %>%
  summarize(count = n(),
            Correct_Glance = sum(Glanced),
            Correct_Called = sum(Called),
            Pct_Correct_Glance = Correct_Glance / count,
            Pct_Correct_Called = Correct_Called /count)

kable(summ_tab,
      caption = 'Summary of data') %>% 
  kable_styling(bootstrap_options = c('striped','hover'))

Summary of data
Error Type	Sign Position	count	Correct_Glance	Correct_Called	Pct_Correct_Glance	Pct_Correct_Called
Gate, No Train	parallel	41	29	5	0.7073171	0.1219512
Gate, No Train	perpendicular	40	7	3	0.1750000	0.0750000
Stuck	parallel	20	2	2	0.1000000	0.1000000
Stuck	perpendicular	20	10	10	0.5000000	0.5000000
Train, No Gate	parallel	41	30	10	0.7317073	0.2439024
Train, No Gate	perpendicular	40	8	5	0.2000000	0.1250000

It seems clear that parallel is the better orientation for both the gate, no train and train, no gate conditions. However, when stuck on the track, it seems better for the sign to be perpendicular to the train track.

gp1 <- 
  ggplot(summ_tab, aes(x = `Error Type`, color = `Sign Position`)) +
  geom_point(aes(y = Pct_Correct_Glance, size = count)) +
  scale_size(range = c(3, 8)) +
  ylim(c(0, 1)) + 
  theme_bw() +
  ggtitle('Percent of correct glances by error type and sign position')


gp2 <-
  ggplot(summ_tab, aes(x = `Error Type`, color = `Sign Position`)) +
  geom_point(aes(y = Pct_Correct_Called, size = count)) +
  scale_size(range = c(3, 8)) +
  ylim(c(0, 1)) + 
  theme_bw() +
  ggtitle('Percent of correct calling behavior by error type and sign position')


ggplotly(gp1)

ggplotly(gp2)

Taking a look at the gender differences:

summ_tab2 <- d %>%
  group_by(Gender, `Error Type`) %>%
  summarize(count = n(),
            Correct_Glance = sum(Glanced),
            Correct_Called = sum(Called),
            Pct_Correct_Glance = Correct_Glance / count,
            Pct_Correct_Called = Correct_Called /count)

gp3 <-
  ggplot(summ_tab2, aes(x = `Error Type`, color = Gender)) +
  geom_point(aes(y = Pct_Correct_Glance, size = count)) +
  scale_size(range = c(3, 8)) +
  ylim(c(0, 1)) + 
  theme_bw() +
  ggtitle('Percent of correct glance behavior by error type and gender')

gp4 <-
  ggplot(summ_tab2, aes(x = `Error Type`, color = Gender)) +
  geom_point(aes(y = Pct_Correct_Called, size = count)) +
  scale_size(range = c(3, 8)) +
  ylim(c(0, 1)) + 
  theme_bw() +
  ggtitle('Percent of correct calling behavior by error type and gender')



ggplotly(gp3)

ggplotly(gp4)

Analysis

Glance analysis

# Intercept = first alphabetical category of error type, namely gate, no train
gm1 <- glm(Glanced ~ `Error Type` + `Sign Position`,
           family = 'binomial',
           data = d)

# Now add interaction
gm2 <- glm(Glanced ~ `Error Type` * `Sign Position`,
           family = 'binomial',
           data = d)

AIC(gm1, gm2) # Second model wins

# Add age and gender to gm2. 
gm3a <- glm(Glanced ~ `Error Type` * `Sign Position` + Age + Gender,
           family = 'binomial',
           data = d)

gm3b <- glm(Glanced ~ `Error Type` * `Sign Position` + Gender,
           family = 'binomial',
           data = d)


AIC(gm2, gm3a, gm3b) # Some improvment, best with age included

tab_model(gm3a)

	Glanced
Predictors	Odds Ratios	CI	p
(Intercept)	9.74	2.63 – 40.21	0.001
`Error Type`Stuck	0.05	0.01 – 0.20	<0.001
`Error Type`Train, No Gate	1.14	0.42 – 3.09	0.801
`Sign Position`perpendicular	0.08	0.02 – 0.22	<0.001
Age	0.98	0.95 – 1.00	0.072
GenderM	0.52	0.25 – 1.06	0.074
GenderTGNC	0.95	0.19 – 5.02	0.948
`Error Type`Stuck:`Sign Position`perpendicular	96.69	13.82 – 961.21	<0.001
`Error Type`Train, No Gate:`Sign Position`perpendicular	1.04	0.23 – 4.82	0.955
Observations	202
R² Tjur	0.316

Interpretation:

Intercept: This is the coefficient for correct glance rate for the alphabetical first category in Error Type, which is ‘Gate, No Train’. Overall, people are likely to glance correctly. The significance is in comparison to the null hypothesis of Odds = 1 for the intercept.

The manual calculations below show how the odds and odds ratios would be calculated for just the Error Type variable alone. The statistical model gm3 above accounts for all the variables. The example below shows how the intercept = the Odds of the first level of the categorical variable, while the other coefficients are the ratios of the odds of that category compared to the intercept.

d_sum <- d %>% 
  group_by(`Error Type`) %>% 
  summarize(count = n(), 
            Correct_Glance = sum(Glanced),
            Pct_Correct_Glance = Correct_Glance / count,
            Odds = ( Correct_Glance / (count - Correct_Glance) ) 
            ) 

kable(d_sum, caption = 'Example calculations of odds') %>% 
  kable_styling(bootstrap_options = c('striped','hover'))

Example calculations of odds
Error Type	count	Correct_Glance	Pct_Correct_Glance	Odds
Gate, No Train	81	36	0.4444444	0.8000000
Stuck	40	12	0.3000000	0.4285714
Train, No Gate	81	38	0.4691358	0.8837209

d_sum$Odds[2] / d_sum$Odds[1]

## [1] 0.5357143

d_sum$Odds[3] / d_sum$Odds[1]

## [1] 1.104651

# Compare to a model.
gm_ex <- glm(Glanced ~ `Error Type`,
           family = 'binomial',
           data = d)
tab_model(gm_ex)

	Glanced
Predictors	Odds Ratios	CI	p
(Intercept)	0.80	0.51 – 1.24	0.318
`Error Type`Stuck	0.54	0.23 – 1.18	0.129
`Error Type`Train, No Gate	1.10	0.59 – 2.05	0.752
Observations	202
R² Tjur	0.016

Compared to Gate, No Train overall, people are much less likely to glance correctly in the Stuck condition
There is no difference in glance rates for Train, No Gate
People are much less likely to glance correctly when the sign is perpendicular to the tracks (but see below…)
Age: Model with age included was better than model without age. However, there is no significant effect of age.
Gender: Males are less than half as likely to glance correctly as females in this study. TGNC individuals show a minor but statistically insignificant increase in correct glance behavior compared to females.
There is an extremely important interaction between sign position and error type: participants are more than 100x more likely to glance correctly when in the stuck position and the sign is perpendicular compared to the intercept
There is also a small but insignificant increase in correct glance behavior when the sign is perpendicular in the Train, No Gate condition.

Call analysis

# Intercept = first alphabetical category of error type, namely gate, no train
cm1 <- glm(Called ~ `Error Type` + `Sign Position`,
           family = 'binomial',
           data = d)

# Now add interaction
cm2 <- glm(Called ~ `Error Type` * `Sign Position`,
           family = 'binomial',
           data = d)

AIC(cm1, cm2) # Second model wins

# Add age and gender to cm2
cm3a <- glm(Called ~ `Error Type` * `Sign Position` + Age + Gender,
           family = 'binomial',
           data = d)

# Without age
cm3b <- glm(Called ~ `Error Type` * `Sign Position` + Gender,
           family = 'binomial',
           data = d)


AIC(cm2, cm3a, cm3b) # cm3b best model (without age)

tab_model(cm3b)

	Called
Predictors	Odds Ratios	CI	p
(Intercept)	0.25	0.08 – 0.64	0.007
`Error Type`Stuck	0.81	0.10 – 4.39	0.814
`Error Type`Train, No Gate	2.49	0.76 – 9.11	0.144
`Sign Position`perpendicular	0.58	0.11 – 2.63	0.487
GenderM	0.23	0.09 – 0.55	0.001
GenderTGNC	1.52	0.30 – 6.76	0.591
`Error Type`Stuck:`Sign Position`perpendicular	20.02	2.09 – 261.83	0.013
`Error Type`Train, No Gate:`Sign Position`perpendicular	0.73	0.10 – 5.54	0.751
Observations	202
R² Tjur	0.164

Interpretation:

Compared to the glance behavior, people are much less likely to make the correct call in the baseline condition of Gate, No Train.
There are no significant effects of Stuck or Train, No Gate on this effect.
Sign position overall does not matter… (but see below)
Males are far less likely to call
There is a 20x increase in correct call behavior when the sign is perpendicular to the tracks when in the Stuck condition

Grade Crossing Sign Glance Analysis

Dan Flynn

3/23/2020

Overview

Variables

EDA

Analysis

Glance analysis

Call analysis